On Effect Size Measures for Nested Measurement Models.

A class of effect size indices are discussed that evaluate the degree to which two nested confirmatory factor analysis models differ from each other in terms of fit to a set of observed variables. These descriptive effect measures can be used to quantify the impact of parameter restrictions imposed in an initially considered model and are free from an explicit relationship to sample size. The described indices represent the extent to which respective linear combinations of the proportions of explained variance in the manifest variables are changed as a result of introducing the constraints. The indices reflect corresponding aspects of the impact of the restrictions and are independent of their statistical significance or lack thereof. The discussed effect size measures are readily point and interval estimated, using popular software, and their application is illustrated with numerical examples.

Comparing Informants for Mental Health Screening at the Preschool Level.

Universal screening for mental health in preschools provides the opportunity for early identification and early intervention, but guidance regarding which informants to use is needed. Preschoolers' (N = 535) parent and teacher reports across two screening forms were analyzed to determine similarities and discrepancies for classification results and screener scores. The analyses also examined if an additional rater provided incrementally valid information to the prediction of longitudinal kindergarten outcomes. Parents' and teachers' screening scores were significantly correlated across forms by rater and across raters. However, categorical classification results indicated that teachers were more likely than parents to rate preschoolers in at-risk ranges across forms. Finally, hierarchical regression analyses revealed teacher ratings were predictive of kindergarten social-emotional, cognitive, and academic outcomes, and that the addition of parent ratings did not significantly improve prediction of outcomes. Implications are discussed in the context of multiple raters within multiple-gating screening procedures.

A randomized controlled trial on the interconnected systems framework for school mental health and PBIS: Focus on proximal variables and school discipline.

This study reviews findings for the first randomized controlled trial (RCT) on the Interconnected Systems Framework (ISF) for school mental health (SMH) and Positive Behavioral Interventions and Supports (PBIS). Since its development in the late 2000s, the ISF has been supported by federally funded centers for SMH and PBIS, and, guided by a national workgroup, is being implemented in >50 communities in the United States. This experimental evaluation of the ISF involved an RCT implemented in 24 schools in two southeastern states, with the ISF implemented in eight schools, PBIS alone implemented in eight schools, and typically co-located PBIS+SMH implemented in eight schools. Related to very poor implementation, documented by two sources of fidelity data, two ISF schools were dropped from major analyses; hence, the study used a treatment on the treated (ToT; Rubin, 1974) as compared to a more traditional Intent-to-Treat approach (ITT; Lachin, 2000). This is the first paper from this large study, with emphasis here on proximal variables and school discipline. Within schools' multi-tiered systems of support (MTSS), ISF schools delivered more Tier 2 (early intervention) and Tier 3 (treatment) interventions to a greater proportion of students than the other two conditions by the second year of the intervention. There was also a dramatic difference in the provision of interventions by community mental health clinicians in ISF schools (almost half of interventions delivered) as compared to PBIS+SMH schools (around 3% of interventions delivered), underscoring the critical role of the ISF in integrating clinicians into MTSS teams and core school functions in SMH. As compared to the other two conditions, ISF schools also had reduced office discipline referrals (ODRs) and in-school suspensions, as well as reduced ODRs and out-of-school suspensions for African American students. Findings are discussed in relation to future directions of education-mental health system partnerships in improving the delivery and impact of SMH programs and services, demonstrated in the ISF.

Design Effect in Multilevel Settings: A Commentary on a Latent Variable Modeling Procedure for Its Evaluation.

A latent variable modeling-based procedure is discussed that permits to readily point and interval estimate the design effect index in multilevel settings using widely circulated software. The method provides useful information about the relationship of important parameter standard errors when accounting for clustering effects relative to conducting single-level analyses. The approach can also be employed as an addendum to point and interval estimation of the intraclass correlation coefficient in empirical research. The discussed procedure makes it easily possible to evaluate the design effect in two-level studies by utilizing the popular latent variable modeling methodology and is illustrated with an example.

Moving towards Implementation of Universal Mental Health Screening by Examining Attitudes toward School-based Practices.

Universal mental health screening is a proactive approach to identify students who may benefit from prevention or early intervention services. Despite known benefits, few schools are engaging in screening efforts and it is critical to examine factors that may impede or enhance implementation. Following implementation of a universal screening program across five preschools and elementary schools, this study investigated the attitudes of teachers (N = 40) and parents (N = 330) and found strong agreement among stakeholders about the acceptability and appropriateness of universal mental health screening. Teachers and parents expressed less willingness to regularly complete screening forms, yet teachers reported that the Behavior Assessment System for Children - Third Edition: Behavioral Emotional Screening System was a usable screening tool. Implications and future directions to enhance implementation efforts are discussed.

A Monte Carlo Study of Confidence Interval Methods for Generalizability Coefficient.

Computing confidence intervals around generalizability coefficients has long been a challenging task in generalizability theory. This is a serious practical problem because generalizability coefficients are often computed from designs where some facets have small sample sizes, and researchers have little guide regarding the trustworthiness of the coefficients. As generalizability theory can be framed to a linear mixed-effect model (LMM), bootstrap and simulation techniques from LMM paradigm can be used to construct the confidence intervals. The purpose of this research is to examine four different LMM-based methods for computing the confidence intervals that have been proposed and to determine their accuracy under six simulated conditions based on the type of test scores (normal, dichotomous, and polytomous data) and data measurement design (p×i×r and p× [i:r]). A bootstrap technique called "parametric methods with spherical random effects" consistently produced more accurate confidence intervals than the three other LMM-based methods. Furthermore, the selected technique was compared with model-based approach to investigate the performance at the levels of variance components via the second simulation study, where the numbers of examines, raters, and items were varied. We conclude with the recommendation generalizability coefficients, the confidence interval should accompany the point estimate.

Evaluating SEM Model Fit with Small Degrees of Freedom.

Research has revealed that the performance of root mean square error of approximation (RMSEA) in assessing structural equation models with small degrees of freedom (df) is suboptimal, often resulting in the rejection of correctly specified or closely fitted models. This study investigates the performance of standardized root mean square residual (SRMR) and comparative fit index (CFI) in small df models with various levels of factor loadings, sample sizes, and model misspecifications. We find that, in comparison with RMSEA, population SRMR and CFI are less susceptible to the effects of df. In small df models, the sample SRMR and CFI could provide more useful information to differentiate models with various levels of misfit. The confidence intervals and p-values of a close fit were generally accurate for all three fit indices. We recommend researchers use caution when interpreting RMSEA for models with small df and to rely more on SRMR and CFI.

Evaluating Restrictive Models in Educational and Behavioral Research: Local Misfit Overrides Model Tenability.

The frequent practice of overall fit evaluation for latent variable models in educational and behavioral research is reconsidered. It is argued that since overall plausibility does not imply local plausibility and is only necessary for the latter, local misfit should be considered a sufficient condition for model rejection, even in the case of omnibus model tenability. The argument is exemplified with a comparison of the widely used one-parameter and two-parameter logistic models. A theoretically and practically relevant setting illustrates how discounting local fit and concentrating instead on overall model fit may lead to incorrect model selection, even if a popular information criterion is also employed. The article concludes with the recommendation for routine examination of particular parameter constraints within latent variable models as part of their fit evaluation.

A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach.

The costs of an objective structured clinical examination (OSCE) are of concern to health profession educators globally. As OSCEs are usually designed under generalizability theory (G-theory) framework, this article proposes a machine-learning-based approach to optimize the costs, while maintaining the minimum required generalizability coefficient, a reliability-like index in G-theory. The authors adopted G-theory parameters yielded from an OSCE hosted by a medical school, reproduced the generalizability coefficients to prepare for optimizing manipulations, applied simulated annealing algorithm to calculate the number of facet levels minimizing the associated costs, and conducted the analysis in various conditions via computer simulation. With a given generalizability coefficient, the proposed approach, virtually an instrument of decision-making supports, found the optimal solution for the OSCE such that the associated costs were minimized. The computer simulation results showed how the cost reductions varied with different levels of required generalizability coefficients. Machine learning-based approaches can be used in conjunction with psychometric modeling to help planning assessment tasks more scientifically. The proposed approach is easy to adopt into practice and customize in alignment with specific testing designs. While these results are encouraging, the possible pitfalls such as algorithmic convergences' failure and inadequate cost assumptions should also be avoided.

Fitting Latent Growth Models with Small Sample Sizes and Non-normal Missing Data.

This study investigates the performance of robust ML estimators when fitting and evaluating small sample latent growth models (LGM) with non-normal missing data. Results showed that the robust ML methods could be used to account for non-normality even when the sample size is very small (e.g., N < 100). Among the robust ML estimators, "MLR" was the optimal choice, as it was found to be robust to both non-normality and missing data while also yielding more accurate standard error estimates and growth parameter coverage. However, the choice "MLMV" produced the most accurate p values for the Chi-square test statistic under conditions studied. Regarding the goodness of fit indices, as sample size decreased, all three fit indices studied (i.e., CFI, RMSEA, and SRMR) exhibited worse fit. When the sample size was very small (e.g., N < 60), the fit indices would imply that a proposed model fit poorly, when this might not be actually the case in the population.

Using Integrative Data Analysis to Investigate School Climate Across Multiple Informants.

Survey research frequently involves the collection of data from multiple informants. Results, however, are usually analyzed by informant group, potentially ignoring important relationships across groups. When the same construct(s) are measured, integrative data analysis (IDA) allows pooling of data from multiple sources into one data set to examine information from multiple perspectives within the same analysis. Here, the IDA procedure is demonstrated via the examination of pooled data from student and teacher school climate surveys. This study contributes to the sparse literature regarding IDA applications in the social sciences, specifically in education. It also lays the groundwork for future educational researchers interested in the practical applications of the IDA framework to empirical data sets with complex model structures.

Using a linear mixed-effect model framework to estimate multivariate generalizability theory parameters in R.

Multivariate generalizability theory (mG-theory) is an important framework in many behavioral and educational studies, as it describes useful psychometric properties of multidimensional assessments. Nevertheless, the use of mG-theory estimation is limited due to the lack of available software for carrying out the necessary calculations: users rely heavily on independent software programs such as mGENOVA and the BUGS/JAGS suite of programs. Considering the prevalence of R software, this paper presents a solution using the glmmTMB package to accomplish the estimation task. Users adopting the proposed method may find it more convenient for conducting both applied investigation and simulation studies without the need to switch between different software programs.

School and Teacher Factors That Promote Adolescents' Bystander Responses to Social Exclusion.

Schools may be one important context where adolescents learn and shape the behaviors necessary for promoting global inclusivity in adulthood. Given the importance of bystanders in halting bullying and peer aggression, the focus of this study is on both moral judgments regarding one type of bullying, social exclusion, and factors that are associated with bystander intervention. The study includes 896 adolescents, who were 6th (N = 450, M age = 11.73), and 9th (N = 446, M age = 14.82) graders, approximately evenly divided by gender. Participants were primarily European-American (63.3%). Results revealed that girls and participants who perceived better relationships between students and teachers were more likely to judge exclusion to be wrong. Further, ethnic minority participants, those who were more anxious about being rejected by their teachers and reported more teacher discrimination were less likely to judge exclusion as wrong. Participants who reported more positive student-teacher relationships, perceptions of a more positive school social environment and more prior experiences of teacher discrimination were more likely to report that they would seek help for the victim. On the other hand, participants who reported being more angry about teacher rejection, experiencing either peer or teacher discrimination, and perceiving they are excluded from opportunities at school were less likely to intervene to come to the aid of a peer who is being excluded. The results document the complex interplay of school and teacher factors in shaping adolescents' bystander responses to social exclusion. Our findings suggest that positive school climate can promote intentions to intervene. However, findings indicate that adolescents who are marginalized in their school environments, and who report experiences of rejection, exclusion or discrimination are not willing or likely to intervene to prevent others from experiencing exclusion.

Fitting Large Factor Analysis Models With Ordinal Data.

A simulation study was conducted to investigate the model size effect when confirmatory factor analysis (CFA) models include many ordinal items. CFA models including between 15 and 120 ordinal items were analyzed with mean- and variance-adjusted weighted least squares to determine how varying sample size, number of ordered categories, and misspecification affect parameter estimates, standard errors of parameter estimates, and selected fit indices. As the number of items increased, the number of admissible solutions and accuracy of parameter estimates improved, even when models were misspecified. Also, standard errors of parameter estimates were closer to empirical standard deviation values as the number of items increased. When evaluating goodness-of-fit for ordinal CFA with many observed indicators, researchers should be cautious in interpreting the root mean square error of approximation, as this value appeared overly optimistic under misspecified conditions.

Scaling the Variance of a Latent Variable While Assuring Constancy of the Model.

This paper investigates how the major outcome of a confirmatory factor investigation is preserved when scaling the variance of a latent variable by the various scaling methods. A constancy framework, based upon the underlying factor analysis formula that enables scaling by modifying components through scalar multiplication, is described; a proof is included to demonstrate the constancy property of the framework. It provides the basis for a scaling method that enables the comparison of the contribution of different latent variables of the same confirmatory factor model to observed scores, as for example, the contributions of trait and method latent variables. Furthermore, it is shown that available scaling methods are in line with this constancy framework and that the criterion number included in some scaling methods enables modifications. The impact of the number of manifest variables on the scaled variance parameter can be modified and the range of possible values. It enables the adaptation of scaling methods to the requirements of the field of application.

Examining the BASC-3 BESS Parent Form-Preschool Using Rasch Methodology.

This study examined the Behavior Assessment System for Children-Third Edition Behavioral and Emotional Screening System, Parent Form-Preschool, which is used to identify children at risk for emotional/behavioral problems. Rasch rating scale methods were used to provide information about psychometric properties of items, parent raters, and the response scale. Using the norming sample of 459 children, we investigated the structure of the screener, performance of the scale, and coverage of the maladaptive behavior construct to identify children with emerging behavioral difficulties. Most items showed good fit to the rating scale methods model with broad coverage of the latent domain. Items measuring internalizing and externalizing problems indicated a greater tendency for at-risk behavior, and items measuring adaptive skills offered a minimal contribution to measuring risk status. No differential item functioning was observed due to child gender; however, differential item functioning was observed between ratings for Caucasian and Hispanic or African American students for select items.

School and Family Factors Predicting Adolescent Cognition Regarding Bystander Intervention in Response to Bullying and Victim Retaliation.

Youth aggression occurs at high rates. Aggressive acts can be curbed through bystander intervention; yet, little is known about school and family factors that predict bystander intervention in response to both aggression and victim retaliation. This research examines school and family factors related to standing up to aggression and intervening before possible retaliation occurs. Participants included 6th and 9th graders (N = 896, 52.8% female), who evaluated how likely they would be to intervene if they observed aggression and if they heard the victim was planning to retaliate. Family and school factors are important predictors of bystander intervention, with higher family management, and more positive school climate associated with greater likelihood of intervention and higher feelings of social exclusion and teacher and peer discrimination associated with inactive responses to aggression and retaliation. Thus, a complex constellation of factors relate to the likelihood of intervening if someone is being victimized or considering retaliation in response to victimization. The results provide guidance and new directions for possible school- and family-based interventions to encourage bystander intervention in instances of aggression.

Evaluating Factorial Invariance: An Interval Estimation Approach Using Bayesian Structural Equation Modeling.

In this study, we introduce an interval estimation approach based on Bayesian structural equation modeling to evaluate factorial invariance. For each tested parameter, the size of noninvariance with an uncertainty interval (i.e. highest density interval [HDI]) is assessed via Bayesian parameter estimation. By comparing the most credible values (i.e. 95% HDI) with a region of practical equivalence (ROPE), the Bayesian approach allows researchers to (1) support the null hypothesis of practical invariance, and (2) examine the practical importance of the noninvariant parameter. Compared to the traditional likelihood ratio test, simulation results suggested that the proposed Bayesian approach could offer additional insight into evaluating factorial invariance, thus, leading to more informative conclusions. We provide an empirical example to demonstrate the procedures necessary to implement the proposed method in applied research. The importance of and influences on the choice of an appropriate ROPE are discussed.